speedup and reduce memory usage of dfa generation
A variety of changes to improve dfa generation
- By switching to Nodevec instead of Node sets we can reduce memory usage slightly and reduce code
- By using charsets for chars we reduce code and increase chances of node merging/reduction which reduces memory usage slightly
- By merging charsets we reduce the number of nodes
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1066
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
Instead of having multiple tables, since we have room post split
of optimization and dump flags just move all the optimization and
dump flags into a common table.
We can if needed switch the flag entry size to a long in the future.
Signed-off-by: John Johansen <john.johansen@canonical.com>
In preparation for more flags (not all of the backend dfa based),
rework the optimization and dump flag handling which has been exclusively
around the dfa up to this point.
- split dfa control and dump flags into separate fields. This gives more
room for new flags in the existing DFA set
- rename DFA_DUMP, and DFA_CONTROL to CONTROL_DFA and DUMP_DFA as
this will provide more uniform naming for none dfa flags
- group dump and control flags into a structure so they can be passed
together.
Signed-off-by: John Johansen <john.johansen@canonical.com>
character sets are just a way of enumerating to exact match rules
more succinctly, so loosen the exact match check to allow them.
Signed-off-by: John Johansen <john.johansen@canonical.com>
The chfa equivalence class shouldn't be a reference. Its needs to
actually exist and be part of the class during later method calls.
As a reference it leads to bad references when used later.
Signed-off-by: John Johansen <john.johansen@canonical.com>
States are not guaranteed to have transitions, but when inserting
a state into the chfa table there is an unconditional dereference
to the states first transition.
This will result in a bad reference and could result in an OOB
flag being set on the state when it shouldn't be.
Fixes: 16b67ddbd ("add ability to use out of band transitions"
Closes: https://gitlab.com/apparmor/apparmor/-/issues/290
Reported-by: Nobel Barakat <nobelbarakat@google.com>
Reported-by: Oleksandr Tymoshenko <ovt@google.com>
Signed-off-by: John Johansen <john.johansen@canonical.com>
The inverse character set lists the characters it doesn't match. If
the inverse character set contains an oob then that is NOT considered
a match. So length should be one.
However because of oobs are handle not containing an oob doesn't mean
there is a match either. Currently the only way to match an oob is
via a positive express (no inverse matches are possible).
Signed-off-by: John Johansen <john.johansen@canonical.com>
The dynamic_cast operator is slow as it needs to look at RTTI
information and even does some string comparisons, especially in deep
hierarchies like the one for Node. Profiling with callgrind showed
that dynamic_cast can eat a huge portion of the running time, as it
takes most of the time that is spent in the simplify_tree()
function. For some complex profiles, the number of calls to
dynamic_cast can be in the range of millions.
This commit replaces the use of dynamic_cast in the Node hierarchy
with a method called is_type(), which returns true if the pointer can
be casted to the specified type. It works by looking at a Node object
field that is an integer with bits set for each type up in the
hierarchy. Therefore, dynamic_cast is replaced by a simple bits
operation.
This change can reduce the compilation times for some profiles more
that 50%, especially in arm/arm64 arch. This opens the door to maybe
avoid "-O no-expr-simplify" in the snapd daemon, as now that option
would make the compilation slower in almost all cases.
This is the example profile used in some of my tests, with this change
the run-time is around 1/3 of what it was before on an x86 laptop:
profile "test" (attach_disconnected,mediate_deleted) {
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.fcitx.Fcitx.InputContext
member="{Close,Destroy,Enable}IC"
peer=(label=unconfined),
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.fcitx.Fcitx.InputContext
member=Reset
peer=(label=unconfined),
dbus receive
bus=fcitx
peer=(label=unconfined),
dbus receive
bus=session
interface=org.fcitx.Fcitx.*
peer=(label=unconfined),
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.fcitx.Fcitx.InputContext
member="Focus{In,Out}"
peer=(label=unconfined),
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.fcitx.Fcitx.InputContext
member="{CommitPreedit,Set*}"
peer=(label=unconfined),
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.fcitx.Fcitx.InputContext
member="{MouseEvent,ProcessKeyEvent}"
peer=(label=unconfined),
dbus send
bus={fcitx,session}
path=/inputcontext_[0-9]*
interface=org.freedesktop.DBus.Properties
member=GetAll
peer=(label=unconfined),
dbus (send)
bus=session
path=/org/a11y/bus
interface=org.a11y.Bus
member=GetAddress
peer=(label=unconfined),
dbus (send)
bus=session
path=/org/a11y/bus
interface=org.freedesktop.DBus.Properties
member=Get{,All}
peer=(label=unconfined),
dbus (receive, send)
bus=accessibility
path=/org/a11y/atspi/**
peer=(label=unconfined),
dbus (send)
bus=system
path=/org/freedesktop/Accounts
interface=org.freedesktop.DBus.Introspectable
member=Introspect
peer=(label=unconfined),
dbus (send)
bus=system
path=/org/freedesktop/Accounts
interface=org.freedesktop.Accounts
member=FindUserById
peer=(label=unconfined),
dbus (receive, send)
bus=system
path=/org/freedesktop/Accounts/User[0-9]*
interface=org.freedesktop.DBus.Properties
member={Get,PropertiesChanged}
peer=(label=unconfined),
dbus (send)
bus=session
interface=org.gtk.Actions
member=Changed
peer=(name=org.freedesktop.DBus, label=unconfined),
dbus (receive)
bus=session
interface=org.gtk.Actions
member={Activate,DescribeAll,SetState}
peer=(label=unconfined),
dbus (receive)
bus=session
interface=org.gtk.Menus
member={Start,End}
peer=(label=unconfined),
dbus (send)
bus=session
interface=org.gtk.Menus
member=Changed
peer=(name=org.freedesktop.DBus, label=unconfined),
dbus (send)
bus=session
path="/com/ubuntu/MenuRegistrar"
interface="com.ubuntu.MenuRegistrar"
member="{Register,Unregister}{App,Surface}Menu"
peer=(label=unconfined),
}
Adjust function and variable names to spell separator correctly. Kept
as a distinct change in case someone wants to cherrypick other fixes.
Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Acked-by: Christian Boltz <apparmor@cboltz.de>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/687
With the exception of the documentation fixes, these should all be
invisible to users.
Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Acked-by: Christian Boltz <apparmor@cboltz.de>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/687
v8 network permissions extend into the range used by exec mapping
so it is important to not blindly do execmapping on both the
file dfa and policydb dfa any more.
Track what type of dfa and its permissions we are building so
we can properly apply exec mapping only when building the
file dfa.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/521
Signed-off-by: John Johansen <john.johansen@canonical.com>
chfa.cc:348:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 16));
^~
chfa.cc:349:3: note: here
case 2:
^~~~
chfa.cc:350:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 8));
^~
chfa.cc:351:3: note: here
case 1:
^~~~
chfa.cc: In function ‘void write_flex_table(std::ostream&, int, Iter, Iter) [with Iter = __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int> >]’:
chfa.cc:348:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 16));
^~
chfa.cc:349:3: note: here
case 2:
^~~~
chfa.cc:350:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 8));
^~
chfa.cc:351:3: note: here
case 1:
^~~~
chfa.cc: In function ‘void write_flex_table(std::ostream&, int, Iter, Iter) [with Iter = __gnu_cxx::__normal_iterator<short unsigned int*, std::vector<short unsigned int> >]’:
chfa.cc:348:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 16));
^~
chfa.cc:349:3: note: here
case 2:
^~~~
chfa.cc:350:4: warning: this statement may fall through [-Wimplicit-fallthrough=]
os.put((char)(*pos >> 8));
^~
chfa.cc:351:3: note: here
case 1:
^~~~
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/561
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <sbeattie@ubuntu.com>
Fixes two resource leaks. https://scan.coverity.com/projects/apparmor
I don't actually know how to link to the individual reports but the
first one comes from an early return. The second comes from an iterator
potentially being empty.
The current encoding makes every xattr optional and uses this to
propogate the permission from the tail to the individual rule match
points.
This however is wrong. Instead change the encoding so that an xattr
(unless optional) is required to be matched before allowing moving
onto the next xattr match.
The permission is carried on the end on each rule portion file match,
xattr 1, xattr 2, ...
Signed-off-by: John Johansen <john.johansen@canonical.com>
xattrs can contain NULL characters in their values which means we can
not user regular NULL transitions to separate values. To fix this
use out of band transition instead.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Currently the NULL character is used as an out of band transition
for string/path elements. This works for them as the NULL character
is not valid for this data. However this does not work for binary
data that can contain a NULL character.
So far we have only dealt with fixed length fields of binary data
making the NULL separator either unnecessary.
However binary data like in the xattr match and mount data field are
variable length and can contain NULL characters. To deal with this
add the ability to specify out of band transitions, that can only
be triggered by code not input data.
The out of band transition can be used to separate variable length
data fields just as the NULL transition has been used to separate
variable length strings.
In the compressed hfa out of band transitions are expressed as a
negative offset from the states base. This leaves us room to expand
the character match range in the future if desired and on average
makes the range between the out of band transition and the input
transitions smaller than would be had if the out of band transition
had been stored after the valid input transitions.
Out of band transitions in the dfa will not break old kernels
that don't know about them, but they won't be able to trigger
the out of band transition match. So they should not be used unless
the kernel indicates that it supports them.
It should be noted that this patch only adds support for a single
out of band transition. If multiple out of band transitions are
required. It is trivial to extend.
- Add a tag indicating support in the kernel
- add a oob max range field to the dfa header so the kernel knows
what the max range that needs verifying is.
- extend oob generation fns to generate oob based on value instead
of a fixed -1.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Currently the parser is not correctly setting the dfa flag value
and it hasn't been caught because base policy uses a flag value
of 0.
Signed-off-by: John Johansen <john.johansen@canonical.com>
As a step in preparing for out of band transitions and double walk
transitions rework the backend from using a char index to a class
with an larger range than char.
Signed-off-by: John Johansen <john.johansen@canonical.com>
When cross compiling apparmor-parser, Makefile will use ar for
creating the static library. However, ar produces libraries on
the build platform. The right ar could be prefixed with the target
platform triples.
Signed-off-by: Xiang Fei Ding <dingxiangfei2009@gmail.com>
Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Ref: https://github.com/NixOS/nixpkgs/pull/63999
Bug: https://gitlab.com/apparmor/apparmor/issues/41
Add userland support for matching based on extended file attributes. This
leverages DFA based matching already in the kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=8e51f908https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=73f488cd
Matching is exposed via flags on the profile:
/usr/bin/* xattrs=(user.foo=bar user.bar=foo) {
# ...
}
xattr values are appended to the existing xmatch via a null transition.
$ echo '/usr/bin/* xattrs=(user.foo=foo user.bar=bar) {}' | \
./parser/apparmor_parser -QT -D expr-tree
DFA: Expression Tree
/usr/bin/[^\0000/]([^\0000/])*(\0000bar)?(\0000foo)?< 0x1>
DFA: Expression Tree
(\a|(\n|(\0002|\t)))< 0x4>
Tested manually on a 4.19 kernel via QEMU+KVM.
TODO:
* ~~Add regression tests~~ (EDIT: done)
* ~~EDIT: add support in the tools~~ (EDIT: done)
Questions for reviewers:
* ~~parser/libapparmor: regex construction probably needs cleaning up~~ (EDIT: done)
* ~~parser/parser_regex.c: confused what xmatch length is for~~ (EDIT: done)
/cc @mjg59
PR: https://gitlab.com/apparmor/apparmor/merge_requests/270
Signed-off-by: John Johansen <john.johansen@canonical.com>
Compiling the parser currently prints a deprecation warning. Remove
throw(int) annotations from function signatures. These aren't required
to catch exceptions.
For example, the following program catches the exception without a
throw(int) annotation:
#include <iostream>
void throw_an_error()
{
throw 3;
return;
}
int main ()
{
try
{
throw_an_error();
}
catch (int e)
{
std::cout << "caught exception " << e << '\n';
}
return 0;
}
This program prints:
$ g++ -o error error.cc
$ ./error
caught exception 3
Signed-off-by: Eric Chiang <ericchiang@google.com>
The length of a xmatch is used to prioritize multiple profiles that
match the same path, with the intent that the more specific match wins.
Currently, the length of a xmatch is computed by the position of the
first regex character.
While trying to work around issues with no_new_privs by combining
profiles, we noticed that the xmatch length computation doesn't work as
expected for multiple regexs. Consider the following two profiles:
profile all /** { }
profile bins /{,usr/,usr/local/}bin/** { }
xmatch_len is currently computed as "1" for both profiles, even though
"bins" is clearly more specific.
When determining the length of a regex, compute the smallest possible
match and use that for xmatch priority instead of the position of the
first regex character.
Expr tree simplification makes multiple passes at simplifying the
expression tree trying to use fatoring rules and heuristics to achieve
the minimum tree, so that dfa construction has fewer nodes to deal
with.
Unfortunately expr tree simplification can slow some policy compiles,
dependent on the type of expressions generated, down, and even worse
is currently subject to never terminating on some expressions as the
left and right passes keep undoing each others work.
Limiting the number of passes that expr tree simplification does can
provide most of its benefits (later passes generally have diminishing
returns), reduces the overhead it has on simple policy where it is of
little benefit, and insures that simplifications can not get stuck in
an infinite loop due to the left and right passes ping-ponging on each
others factoring.
Note: This also results in a performance improvement in evince
compiles, and general policy compiles because it achieves a better
balance between time spent on simplifying the tree to remove nodes and
time the dfa build requires to build with extra nodes and then
eliminate with minimization.
$ time apparmor_parser -QT /etc/apparmor.d/usr.bin.evince
real 0m2.744s
user 0m2.714s
sys 0m0.028s
vs.
$ time apparmor_parser -QT /etc/apparmor.d/usr.bin.evince
real 0m2.992s
user 0m2.979s
sys 0m0.012s
and
$ time apparmor_parser -QT /etc/apparmor.d/
real 0m3.568s
user 0m14.529s
sys 0m0.152s
vs.
$ time apparmor_parser -QT /etc/apparmor.d/
real 0m3.741s
user 0m15.400s
sys 0m0.179s
PR: https://gitlab.com/apparmor/apparmor/merge_requests/246
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Seth Arnold <seth.arnold@canonical.com>
Elaborate in class comment of firstpos, lastpos, followpos, and nullable
fields beyond just referencing the Dragon book. Also add the section of
the book these are explained in.
The current rule simplification algorithm has issues that need to be
addressed in a rewrite, but it is still often a win, especially for
larger profiles.
However doing rule simplification as a single pass limits what it can
do. We default to right simplification first because this has historically
shown the most benefits. For two reasons
1. It allowed better grouping of the split out accept nodes that we
used to do (changed in previous patches)
2. because trailing regexes like
/foo/**,
/foo/**.txt,
can be combined and they are the largest source of node set
explosion.
However the move to unique node sets, eliminates 1, and forces 2 to
work within only the single unique permission set on the right side
factoring pass, but it still incures the penalty of walking the whole
tree looking for potential nodes to factor.
Moving tree simplification into the construction phases gets rid of
the need for the right side factoring pass to walk other node sets
that will never combine, and since we are doing simplification we can
do it before the cat and permission nodes are added reducing the
set of nodes to look at by another two.
We do loose the ability to combine nodes from different sets during
the left factoring pass, but experimentation shows that doing
simplification only within the unique permission sets achieve most of
the factoring that a single global pass would achieve.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
Currently rules are added to the expression tree in order, and then
tree simplification and factoring is done. This forces simplification
to "search" through the tree to find rules with the same permissions
during right factoring, which dependent on ordering of factoring may
not be able to group all rules of the same permissions.
Instead of having tree factoring do the work to regroup rules with the
same permissions, pregroup them as part of the expr tree construction.
And only build the full tree when the dfa is constructed.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
accept nodes per perm bit where done from the very begining in a
false belief that they would help produce minimized dfas because
a nfa states could share partial overlapping permissions.
In reality they make tree factoring harder, reduce in longer nfa
state sets during dfa construction and do not result in a minimized
dfa.
Moving to unique permission sets, allows us to minimize the number
of nodes sets, and helps reduce recreating each set type multiple
times during the dfa construction.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
This fixes the incorrect compilation of audit modifiers for exec and
pivot_root as detailed in
https://launchpad.net/bugs/1431717https://launchpad.net/bugs/1432045
The permission accumulation routine on the backend was incorrectly setting
the audit mask based off of the exec type bits (info about the exec) and
not the actual exec permission.
This bug could have also caused permissions issues around overlapping exec
generic and exact match exec rules, except the encoding of EXEC_MODIFIERS
ensured that the
exact_match_allow & AA_USER/OTHER_EXEC_TYPE
test would never fail for a permission accumulation with the exec permission
set.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
This patch adjusts the bison grammer in libapparmor and the parser
to use the %define api.pure directive instead of the deprecated
%pure_parser and %pure-parser keywords. Bison had been warning about
the former:
libraries/libapparmor/src/grammar.y:71.1-12: warning: deprecated directive, use ‘%pure-parser’ [-Wdeprecated]
%pure_parser
^^^^^^^^^^^^
Signed-off-by: Steve Beattie <steve@nxnw.org>
Acked-by: Seth Arnold <seth.arnold@canonical.com>
Refactor add_new_state into two versions, one that splits anodes from
nnodes, and one for use when anodes and nnodes are presplit
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
Acked-by: Seth Arnold <seth.arnold@canonical.com>
The shared node type will be used in the future to add new capabilities
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
We need to rework permission type mapping to nodesets, which means we
need to move the nodeset computations earlier in the dfa creation
processes, instead of a post step of follow(), so move the nodeset
into expr-tree
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
Acked-by: Seth Arnold <seth.arnold@canonical.com>
Also for characters that are not recognized as a valid escape seq
make sure that the character is emitted.
previously
\$ resulted in \
where it should have been \$ if $ wasn't recognized
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
This cleans things up a bit and fixes a bug where not all rules are
getting properly counted so that the addition of policy_mediation
rules fails to generate the policy dfa in some cases.
Because the policy dfa is being generated correctly now we need to
fix some tests to use the new -M flag to specify the expected features
set of the test.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Steve Beattie <steve@nxnw.org>
Acked-by: Seth Arnold <seth.arnold@canonical.com>
