first hash does hashing on state just state transitions, which always results
in a performance improvement.
The second does hashing based off of accept permissions, which can create
more initial states but can result in not being able to achieve a true
minimum dfa. This can also lead to slowing down total dfa creation because
while minimization, compression can take longer if the dfa isn't completely
minimized.
permission hashing is currently required, as minimization does not accumulate
redundant Node permissions.
memory than just using the NodeSet size to short circuit comparison but it
improves on the case where compared sets have the same size. It is possible
that this will slow down small dfa generation slightly but the trade off for
large dfa's (which are the slow ones to generate) is worth it.
This results in another performance bump over using the NodeSize is NodeSet
comparison, and the amount of improvement increases with larger dfas
of pointers when it isn't necessary. This results in a nice little
performance increase in dfa creation.
This is more of a proof of concept patch, and is replaced by the next
patch which does better short circuiting via hashing
the large side, and I experimented with different ways to split this up but in
the end, anything I could do would result in a series of dependent patches
that would require all of them to be applied to get meaningful functional
changes.
The patch structural reworks the dfa so that
- there is a new State class, it takes the place of sets of nodes in the
dfa, and allows storing state information within the state
- removes the dfa transition table, which mapped sets of nodes to a
transition table, by moving the transition into the new state class
- computes dfa state permissions once (stored in the state)
- expression tree nodes are independent from a created dfa. This allows
computed expression trees, and sets of Nodes (used as protostates when
computing the dfa). To be managed independent of the dfa life time.
This will allow reducing the amount of memory used, in the future,
and will also allow separating the expression tree logic out into
its own file.
The patch has some effect on reducing peak memory usage, and computation
time. The actual amount of reduction is dependent on the number of states
in the dfa with larger saving being achieved on larger dfas. Eg. for
the test evince profile I was using it makes the parser about 7% faster with a
peak memory usage about 12% less.
This patch changes the initial partition hashing of minimization resulting
in slightly smaller dfas.
tree. It is limited in that it doesn't currently handle the permissions of a rule.
conversion output presents an aare -> prce conversion followed by 1 or more expression
tree rules, governed by what the rule does.
eg.
aare: /** -> /[^/\x00][^\x00]*
rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])*
eg.
echo "/foo { /** rwlkmix, } " | ./apparmor_parser -QT -D rule-exprs -D expr-tree
aare: /foo -> /foo
aare: /** -> /[^/\x00][^\x00]*
rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])*
rule: /[^/\x00][^\x00]*\x00/[^/].* -> /[^\0000/]([^\0000])*\0000/[^/](.)*
DFA: Expression Tree
(/[^\0000/]([^\0000])*(((((((((((((<513>|<2>)|<4>)|<8>)|<16>)|<32>)|<64>)|<8404992>)|<32768>)|<65536>)|<131072>)|<262144>)|<524288>)|<1048576>)|/[^\0000/]([^\0000])*\0000/[^/](.)*((<16>|<32>)|<262144>))
This simple example shows many things
1. The profile name under goes pcre conversion. But since no regular expressions where found
it doesn't generate any expr rules
2. /** is converted into the pcre expression /[^\0000/]([^\0000])*
3. The pcre expression /[^\0000/]([^\0000])* is converted into two rules that are then
converted into expression trees.
The reason for this can not be seen by the output as this is actually triggered by
permissions separation for the rule. In this case the link permission is separated
into what is shown as the second rule: statement.
4. DFA: Expression Tree dump shows how these rules are combined together
You will notice that the rule conversion statement is fairly redundant currently as it just
show pcre to expression tree pcre. This will change when direct aare parsing occurs,
but currently serves to verify the pcre conversion step.
It is not the prettiest patch, as its touching some ugly code that is schedule to be cleaned
up/replaced. eg. convert_aaregex_to_pcre is going to replaced with native parse conversion
from an aare straight to the expression tree, and dfaflag passing will become part of the
rule set.
It changes the table resizing so that there is always sufficient
high entries in the table, preventing bounds violations from
occurring.
Previously the resize allocation was always based on the character
set range for a state, which could be more or less than actually
required, and packing would waste some space when over allocation
was done.
As a result this patch in general results in slightly smaller
transition tables even though it enforcing the minimum required
padding to avoid bounds violations.
are computed and stored in a map, that is not cleaned up. This means that the labeling
is retained across different dfas.
Move the labeling into expr node as this takes less memory than using a map and will
also separates node labeling so its per dfa instead of global. In addition this means
the labeling is cleanedup/freed when the expr tree is freed without any extra work.
each expression tree node and then used as input to create the dfa states.
Currently they are not being freed until the nodes are destroyed, but the information
is no longer needed once the dfa has been created. Cleaning them up early reduces
peak memory usage.
imediately after the current partition being considered, instead of
at the back of the parition list. This does two things, it makes it
more likely the data is in cache, and it also in general results in
more partitions being created in a single pass.
not an algorithmic improvement. It does the same basic algorithm of
test until it can insert the data, but instead of only tracking the
first free entry (and recomputing it each pass). It tracks all
free entries reducing the number of comparisons done and the table
grows in size.
This may actually result in a small loss on small tables, but is a win
for larger tables.
Update the hash calculation to guarentee that states with a different
number of transition entries will be placed in seperate partitions.
This will allow for a better character transition based state comparison.
Add basic Hopcroft based dfa minimization. It currently does a simple
straight state comparison that can be quadratic in time to split partitions.
This is offset however by using hashing to setup the initial partitions so
that the number of states within a partition are relative few.
The hashing of states for initial partition setup is linear in time. This
means the closer the initial partition set is to the final set, the closer
the algorithm is to completing in a linear time. The hashing works as
follows: For each state we know the number of transitions that are not
the default transition. For each of of these we hash the set of letters
it can transition on using a simple djb2 hash algorithm. This creates
a unique hash based on the number of transitions and the input it can
transition on. If a state does not have the same hash we know it can not
the same as another because it either has a different number of transitions
or or transitions on a different set.
To further distiguish states, the number of transitions of each transitions
target state are added into the hash. This serves to further distiguish
states as a transition to a state with a different number of transitions
can not possibly be reduced to an equivalent state.
A further distinction of states is made for accepting states in that
we know each state with a unique set of accept permissions must be in
its own partition to ensure the unique accept permissions are in the
final dfa.
The unreachable state removal is a basic walk of the dfa from the start
state marking all states that are reached. It then sweeps any state not
reached away. This does not do dead state removal where a non accepting
state gets into a loop that will never result in an accepting state.
This will allow turning on and off various debug dumps as needed.
Multiple dump options can be specified as needed by using multiple
options.
eg. apparmor_parser -D variables
apparmor_parser -D dfa-tree -D dfa-simple-tree
The help option has also been updated to take an optional argument
to display help about give parameters, currently only dump is supported.
eg. apparmor_parser -h # standard help
apparmor_parser -h=dump # dump info about --dump options
Also Enable the dfa expression tree dumps
Change the globbing conversion to include [^\x00]. This reduces cases of
artifical overlap between globbing rules, and link rules. Link rules
are encoded to use a \0 char to seperate the 2 match parts of the rule.
Before this fix a glob * or ** could match against the \0 seperator
resulting the generation of dfa states for that overlap. This of course
can never happen as \0 is not a valid path name character.
In one example stress policy when adding the rule
owner /** rwl,
this change made the difference between having a dfa with 55152 states
and one with 30019
- disable charter, charset merging. This can actually hamper optimization
in some cases and needs special cases added to the factoring code.
The charset code is merged off into its own routines that can be
reenabled at a later time.
- fix a couple bugs in tree simplifications that would cause earlier
exit before the tree had even reached a local minima
I particular the t != c portion of the simplify_tree, would cause
the loop to exit early if it didn't change but other modifications
had been made.
- remove the extra epsnode that was getting added to the created tree
- optimize the forward factor alt loop so that it will find all left
factor matches down the alt subtree without having to loop and recompare
against nodes that were already checked
These changes result in small improvements in most cases, but in some
policies the changes result in very large wins. The early bailout of
optimizations was causing 2.5* as many dfa states in one particular
stress test policy.
been made but only from the top level. This allows us to get the
optimizations that were missed, while not causing the massive recursive call
explosion we had before.
Apply tree factoring and simplification techniques to reduce the number of
states used in computing the dfa. This can have an exponential impact
on both space and time for dfa generation.
