eMedia Hosting

will take a bit to migrate it but will get rid of those containers

I'm launching a different server for all TestU stuff

ok

Ian we are shutting down a runaway instance of m38 testu77

951 enterme+ 20 0 70732 57900 4552 R 99.7 0.1 0:44.59 pdftoppm -f 0 -l 0 -aa no -aaVector no -rx 300 -ry 300 -png /opt/entermediadb/webapp/WEB-INF/data/testu/system/originals/Legacy/1100/36___121.pdf poppler
2360 root 20 0 112680 57432 57104 S 0.3 0.1 0:12.72 /usr/lib/systemd/systemd-journald
34222 enterme+ 20 0 68252 55132 4484 R 100.0 0.1 0:35.90 pdftoppm -f 0 -l 0 -aa no -aaVector no -rx 300 -ry 300 -png /opt/entermediadb/webapp/WEB-INF/data/testu/system/originals/Legacy/1158/76___121.pdf poppler
6625 root 20 0 2673664 54248 15384 S 0.0 0.1 1:24.72 /usr/bin/containerd
1951 enterme+ 20 0 57344 43940 4816 R 100.0 0.0 43:41.15 pdftoppm -f 0 -l 0 -aa no -aaVector no -rx 300 -ry 300 -png /opt/entermediadb/webapp/WEB-INF/data/testu/system/originals/Legacy/38/FCOM 19 Mar 2018 Complete 2.pdf poppler
6603 root 20 0 289948 35028 31452 S 0.0 0.0 0:08.38 /usr/sbin/rsyslogd

And try to get a rsync while it's up just in case

I will prune some stuff I don't really need anyway, keep it to bare min

ok

Its likely a power supply issue. I will have them try and find the issue. But likely this server is dead for now

I will wait a few min to start docker? See if it crashes independant of that?

I'm on agian

I wonder if there are any crash logs or anything. Docker shouldn't take down the entire server

ok

It shows a screen shot of the server booting now

CAn you login to ssh via that console or anything?

certainly seems different than anything I've seen before

I will have them look deeper

We have a console login for it and it says its on: Host is currently on

I remounted ZFS, started docker and everything looked fine
then it stopped again

I really don't know

Some kind of hardware problem?

And now I can't get in again

ok done, it should recover now

I can fix the mount one sec

I see this in the boot log: [ OK ] Started Load Kernel Modules.
Starting Apply Kernel Variables...
Starting Configure read-only root support...
Starting Rebuild Hardware Database...
Activating swap /swapfile...
Starting Flush Journal to Persistent Storage...
Starting Mount ZFS filesystems...
Starting udev Kernel Device Manager...
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Configure read-only root support.
[FAILED] Failed to activate swap /swapfile.
See 'systemctl status swapfile.swap' for details.
[DEPEND] Dependency failed for Swap.
Starting Load/Save Random Seed...
[FAILED] Failed to start Mount ZFS filesystems.
See 'systemctl status zfs-mount.service' for details.
[ OK ] Started Load/Save Random Seed.

yeah it created a ton of empty dockers

I think I can fix it, there is a script here

Soemthign is still wrong, I think the drive didn't mount
ZFS

it's been running fine

I hope we dont have a hardware issue

no just certificates and upgrades

So nothin weird there

root: cd
990 cd
991 cat ~/.bash_history
992 exit
993 exit
994 history
995 exit
996 ip addr
997 ip addr | less
998 ping 8.8.8.8
999 exit

Last history logs look like this: wget -O http://dev.ijsolutions.ca/jenkins/jo/colony42/lastSuccessfulBuild/artifact/deploy/u[A
990 nano olw.sh
991 ./olw.sh
992 cat upgrade.xml
993 clear
994 ./logs.sh
995 ./olw.sh
996 cat upgrade.xml
997 ./olw.sh
998 sudo cerbot --nginx -d oneliveweb.com
999 sudo certbot --redirect --nginx -d oneliveweb.com
1000 sudo certbot --nginx -d beta.testu.co

must have, I just noticed and couldn't ssh in

Perhaps it crashed

They said it was a software issue. Why did it get shut down?

I can get rid of a few but I don't think that was the cause

I got rid of a slew of them the other day - That won't be the problem, it's been running like this for at least 2-3 years

Can we remove some of these docker containers?
M38

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32298cc8bf90 entermediadb/entermedia10:latest "/usr/bin/entermedia…" 6 days ago Up 8 minutes colony4242
40977a5d4f68 1b73f2d79127 "/usr/bin/entermedia…" 3 weeks ago Up 8 minutes cheese67
dd40f179557e 1b73f2d79127 "/usr/bin/entermedia…" 3 weeks ago Up 8 minutes 0.0.0.0:1990->1935/tcp, :::1990->1935/tcp paes.testu.co90
65d88d160902 1b73f2d79127 "/usr/bin/entermedia…" 3 weeks ago Up 8 minutes 0.0.0.0:1983->1935/tcp, :::1983->1935/tcp demotestu83
4b47efae42be 1b73f2d79127 "/usr/bin/entermedia…" 3 weeks ago Up 8 minutes 0.0.0.0:1978->1935/tcp, :::1978->1935/tcp testu-dev78
60dae95aab47 1b73f2d79127 "/usr/bin/entermedia…" 3 weeks ago Up 8 minutes 0.0.0.0:1960->1935/tcp, :::1960->1935/tcp oneliveweb60
1c0aac6d54e1 jenkins/jenkins:lts "/usr/bin/tini -- /u…" 4 weeks ago Restarting (1) 28 seconds ago oneliveweb25
e2d6bb2ab055 990f0df788a1 "/usr/bin/entermedia…" 8 months ago Up 8 minutes willow65
851fb91829a2 1a958a4539d4 "/usr/bin/entermedia…" 8 months ago Up 8 minutes evolvedlearning10
11406e94554d entermediadb/entermediadb9:latest "/usr/bin/entermedia…" 12 months ago Up 8 minutes connect53
3cba3983389d c747f15bf8e7 "/usr/bin/entermedia…" 17 months ago Up 8 minutes netevolved55
a752d3d00110 993aa438e36b "/usr/bin/entermedia…" 17 months ago Up 8 minutes testu77
8b46e0a3901c 29d4fb0c3848 "/usr/bin/entermedia…" 18 months ago Up 8 minutes testreadypro11
02e054a1c427 ed7fa90d9dde "/usr/bin/entermedia…" 3 years ago Up 8 minutes agbu99
23b0ac80785f gcr.io/google-containers/cadvisor "/usr/bin/cadvisor -…" 3 years ago Up 8 minutes (healthy) 0.0.0.0:9110->8080/tcp, :::9110->8080/tcp cadvisor
ba7c96a921fb 10f2ac0d4464 "/usr/bin/entermedia…" 3 years ago Up 8 minutes metroeastb181
50e77dfdd5a7 10f2ac0d4464 "/usr/bin/entermedia…" 3 years ago Up 8 minutes almb179
195c0fd9272b f53a12974117 "/usr/bin/entermedia…" 4 years ago Up 8 minutes dpci285
ad52fa89164e da3396d818b7 "/usr/bin/entermedia…" 4 years ago Up 8 minutes alm_dev102
78454a245732 da3396d818b7 "/usr/bin/entermedia…" 4 years ago Up 8 minutes metroeast100