[ceph-users] MGR Dashboard

[Ashley Merrick]

Hey,

After rebooting a server that hosts the MGR Dashboard I am now unable to
get the dashboard module to run.

Upon restarting the mgr service I see the following :

ImportError: No module named ordered_dict
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE
Serving on http://:::9283
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE Bus
STARTED


I have checked using pip install ordereddict and it states the module is
already installed.

[Jos Collin]

http://tracker.ceph.com/issues/19913

Post by Ashley Merrick
Hey,
After rebooting a server that hosts the MGR Dashboard I am now unable
to get the dashboard module to run.
ImportError: No module named ordered_dict
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14]
ENGINE Serving on http://:::9283
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14]
ENGINE Bus STARTED
I have checked using pip install ordereddict and it states the module
is already installed.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Lenz Grimmer]

Hi Ashley,

Post by Ashley Merrick
After rebooting a server that hosts the MGR Dashboard I am now unable to
get the dashboard module to run.
ImportError: No module named ordered_dict
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE
Serving on http://:::9283
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE
Bus STARTED
I have checked using pip install ordereddict and it states the module is
already installed.

What version of Ceph is this? What OS?

Lenz

[Ashley Merrick]

Hey,

Sorry missed the basic info!!

Latest Mimic 13.2.2

Ubuntu 18.04

,Ashley

Post by Lenz Grimmer
Hi Ashley,

Post by Ashley Merrick
After rebooting a server that hosts the MGR Dashboard I am now unable to
get the dashboard module to run.
ImportError: No module named ordered_dict
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE
Serving on http://:::9283
Nov 29 07:13:14 ceph-m01 ceph-mgr[12486]: [29/Nov/2018:07:13:14] ENGINE
Bus STARTED
I have checked using pip install ordereddict and it states the module is
already installed.

What version of Ceph is this? What OS?
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG NÌrnberg)
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Lenz Grimmer]

Post by Ashley Merrick
Sorry missed the basic info!!
Latest Mimic 13.2.2
Ubuntu 18.04

Thanks. So it worked before the reboot and did not afterwards? What
changed? Did you perform an OS update?

Would it be possible for you to paste the entire mgr log file messages
that are printed after the manager restarted? Have you tried to
explicitly enable the dashboard by running "ceph mgr module enable
dashboard"?

Lenz

[Ashley Merrick]

Yeah had a few OS updates, but not related directly to CEPH.

The full error log after a reboot is :

2018-11-29 11:24:22.494 7faf046a1700 1 mgr[restful] server not running: no
certificate configured
2018-11-29 11:24:22.586 7faf05ee4700 -1 log_channel(cluster) log [ERR] :
Unhandled exception from module 'dashboard' while running on mgr.ceph-m01:
No module named ordered_dict
2018-11-29 11:24:22.586 7faf05ee4700 -1 dashboard.serve:
2018-11-29 11:24:22.586 7faf05ee4700 -1 Traceback (most recent call last):
File "/usr/lib/ceph/mgr/dashboard/module.py", line 276, in serve
mapper = generate_routes(self.url_prefix)
File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 118, in
generate_routes
ctrls = load_controllers()
File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 73, in
load_controllers
package='dashboard')
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/lib/ceph/mgr/dashboard/controllers/rgw.py", line 10, in
<module>
from ..services.rgw_client import RgwClient
File "/usr/lib/ceph/mgr/dashboard/services/rgw_client.py", line 5, in
<module>
from ..awsauth import S3Auth
File "/usr/lib/ceph/mgr/dashboard/awsauth.py", line 49, in <module>
from requests.auth import AuthBase
File "/usr/lib/python2.7/dist-packages/requests/__init__.py", line 97, in
<module>
from . import utils
File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 26, in
<module>
from ._internal_utils import to_native_string
File "/usr/lib/python2.7/dist-packages/requests/_internal_utils.py", line
11, in <module>
from .compat import is_py2, builtin_str, str
File "/usr/lib/python2.7/dist-packages/requests/compat.py", line 47, in
<module>
from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict


I have tried "ceph mgr module enable dashboard" and it says already
enabled, I tried a disable restart and enable and get the same error above.

,Ashley

Post by Lenz Grimmer

Post by Ashley Merrick
Sorry missed the basic info!!
Latest Mimic 13.2.2
Ubuntu 18.04

Thanks. So it worked before the reboot and did not afterwards? What
changed? Did you perform an OS update?
Would it be possible for you to paste the entire mgr log file messages
that are printed after the manager restarted? Have you tried to
explicitly enable the dashboard by running "ceph mgr module enable
dashboard"?
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG NÌrnberg)

[Ashley Merrick]

Managed to fix the issue with some googling from the error above.

There is a bug with urllib3 1.24.1 which breaks the module ordered_dict (1)

I rolled back to a working version "pip install urllib3==1.23" and
restarted the mgr service and all is now working.

Thanks,
Ashley

(1)https://github.com/urllib3/urllib3/issues/1456

Post by Ashley Merrick
Yeah had a few OS updates, but not related directly to CEPH.
no certificate configured
No module named ordered_dict
File "/usr/lib/ceph/mgr/dashboard/module.py", line 276, in serve
mapper = generate_routes(self.url_prefix)
File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 118, in
generate_routes
ctrls = load_controllers()
File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 73, in
load_controllers
package='dashboard')
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in
import_module
__import__(name)
File "/usr/lib/ceph/mgr/dashboard/controllers/rgw.py", line 10, in
<module>
from ..services.rgw_client import RgwClient
File "/usr/lib/ceph/mgr/dashboard/services/rgw_client.py", line 5, in
<module>
from ..awsauth import S3Auth
File "/usr/lib/ceph/mgr/dashboard/awsauth.py", line 49, in <module>
from requests.auth import AuthBase
File "/usr/lib/python2.7/dist-packages/requests/__init__.py", line 97,
in <module>
from . import utils
File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 26, in
<module>
from ._internal_utils import to_native_string
File "/usr/lib/python2.7/dist-packages/requests/_internal_utils.py",
line 11, in <module>
from .compat import is_py2, builtin_str, str
File "/usr/lib/python2.7/dist-packages/requests/compat.py", line 47, in
<module>
from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict
I have tried "ceph mgr module enable dashboard" and it says already
enabled, I tried a disable restart and enable and get the same error above.
,Ashley

Post by Lenz Grimmer

Post by Ashley Merrick
Sorry missed the basic info!!
Latest Mimic 13.2.2
Ubuntu 18.04

Thanks. So it worked before the reboot and did not afterwards? What
changed? Did you perform an OS update?
Would it be possible for you to paste the entire mgr log file messages
that are printed after the manager restarted? Have you tried to
explicitly enable the dashboard by running "ceph mgr module enable
dashboard"?
Lenz
--
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG NÌrnberg)

[Lenz Grimmer]

Hi Ashley,

Post by Ashley Merrick
Managed to fix the issue with some googling from the error above.
There is a bug with urllib3 1.24.1 which breaks the module ordered_dict (1)

Good spotting!

Post by Ashley Merrick
I rolled back to a working version "pip install urllib3==1.23" and
restarted the mgr service and all is now working.

Glad to hear you got it working again. Thanks for the update!

Post by Ashley Merrick
(1)https://github.com/urllib3/urllib3/issues/1456

Lenz

[Lenz Grimmer]

Post by Ashley Merrick
Yeah had a few OS updates, but not related directly to CEPH.

But they seem to be the root cause of the issue you're facing. Thanks
for sharing the entire log entry.

Post by Ashley Merrick
no certificate configured
Unhandled exception from module 'dashboard' while running on
mgr.ceph-m01: No module named ordered_dict
  File "/usr/lib/ceph/mgr/dashboard/module.py", line 276, in serve
    mapper = generate_routes(self.url_prefix)
  File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 118,
in generate_routes
    ctrls = load_controllers()
  File "/usr/lib/ceph/mgr/dashboard/controllers/__init__.py", line 73,
in load_controllers
    package='dashboard')
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/lib/ceph/mgr/dashboard/controllers/rgw.py", line 10, in
<module>
    from ..services.rgw_client import RgwClient
  File "/usr/lib/ceph/mgr/dashboard/services/rgw_client.py", line 5, in
<module>
    from ..awsauth import S3Auth
  File "/usr/lib/ceph/mgr/dashboard/awsauth.py", line 49, in <module>
    from requests.auth import AuthBase
  File "/usr/lib/python2.7/dist-packages/requests/__init__.py", line 97,
in <module>
    from . import utils
  File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 26, in
<module>
    from ._internal_utils import to_native_string
  File "/usr/lib/python2.7/dist-packages/requests/_internal_utils.py",
line 11, in <module>
    from .compat import is_py2, builtin_str, str
  File "/usr/lib/python2.7/dist-packages/requests/compat.py", line 47,
in <module>
    from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict
I have tried "ceph mgr module enable dashboard" and it says already
enabled, I tried a disable restart and enable and get the same error above.

Try re-installing the following packages: "python-urllib3" and
"python-requests" via apt - somehow Python fails to import a method from
the former library.

Lenz