lxc filling /dev

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

lxc filling /dev

William Kenworthy
What would cause an lxc instance to leak memory filling /dev?  Of
course, when it gets to 100% things ... stop! - but only for that lxc
instance.

The host is an odroid N2 with a gentoo-sources kernel and a gentoo arm
(aarch64) userland running 6 gentoo based lxc instances.

2 ~ # lxc-attach -n mail -- bash -c "df -h"
Filesystem                      Size  Used Avail Use% Mounted on
/dev/root                       115G   45G   64G  42% /
none                            492K  320K  172K  66% /dev
mfsmaster.san.localdomain:9421   17T  7.5T  8.9T  46% /usr/portage
none                            1.8G     0  1.8G   0% /dev/shm
tmpfs                           350M  148K  349M   1% /run
cgroup_root                      10M     0   10M   0% /sys/fs/cgroup
n2 ~ #

du and ls -al do not give any clues, the host /dev is normal and all
running lxc instances do it, but at different rates


BillK


Reply | Threaded
Open this post in threaded view
|

Re: lxc filling /dev

Rich Freeman
On Sun, Feb 16, 2020 at 7:57 PM William Kenworthy <[hidden email]> wrote:
>
> 2 ~ # lxc-attach -n mail -- bash -c "df -h"
> none                            492K  320K  172K  66% /dev
> du and ls -al do not give any clues, the host /dev is normal and all
> running lxc instances do it, but at different rates

Are you running ls -al from INSIDE the container?  If you're running
it on the host you won't see anything because it is almost certainly
in a separate mount namespace, and so it is invisible from the host.
In particular, any files you see in rootdir/dev from the host are NOT
visible in the container, and vice-versa.

I don't use lxc, but if I had to take a wild guess your /dev isn't
being properly initialized inside, and some typical device node is
being created as a regular file and stuff like "echo foo > /dev/null"
is actually writing to a real file there, filling up the tmpfs.

Try:
lxc-attach -n mail -- bash -c "ls -l --recursive /dev"

Or launch an interactive shell inside the container and just poke
around in there.  I have no idea what the "lxc way" to launch a shell
is, but you can always use:
nsenter --target <pid> --all /bin/bash
(where <pid> is the pid on the host of a process inside the container)

nsenter is part of util-linux

--
Rich

Reply | Threaded
Open this post in threaded view
|

Re: lxc filling /dev

William Kenworthy
Thank Rich,

    It seems to be tty12 (console logging) - I think disabling it in
syslog-ng will be easiest but will do some testing first.

The recursive switch shows tty12 regularly ticking up.

BillK


On 17/2/20 10:13 am, Rich Freeman wrote:

> On Sun, Feb 16, 2020 at 7:57 PM William Kenworthy <[hidden email]> wrote:
>> 2 ~ # lxc-attach -n mail -- bash -c "df -h"
>> none                            492K  320K  172K  66% /dev
>> du and ls -al do not give any clues, the host /dev is normal and all
>> running lxc instances do it, but at different rates
> Are you running ls -al from INSIDE the container?  If you're running
> it on the host you won't see anything because it is almost certainly
> in a separate mount namespace, and so it is invisible from the host.
> In particular, any files you see in rootdir/dev from the host are NOT
> visible in the container, and vice-versa.
>
> I don't use lxc, but if I had to take a wild guess your /dev isn't
> being properly initialized inside, and some typical device node is
> being created as a regular file and stuff like "echo foo > /dev/null"
> is actually writing to a real file there, filling up the tmpfs.
>
> Try:
> lxc-attach -n mail -- bash -c "ls -l --recursive /dev"
>
> Or launch an interactive shell inside the container and just poke
> around in there.  I have no idea what the "lxc way" to launch a shell
> is, but you can always use:
> nsenter --target <pid> --all /bin/bash
> (where <pid> is the pid on the host of a process inside the container)
>
> nsenter is part of util-linux
>