Way over thinking it. It's not hazardous it even says it's stable and non hazardous in the document you provided and can even be disposed of exactly like any other non flammable trash when done.
It's a desiccant so it's an irritant cause it can dry your skin or eyes. And it's a very strong desiccant that absorbs water very well.
Silica actually has the same warning about washing your skin and avoiding dust. Same reasons just one is slightly better at it.
Activated Alumina is actually already used in lots of things you may interact with like industrial water filters or whole house ones, it's used to clean cannabis and vape oil, and it's even used in prosthetics.
You are way overthinking it. Like California putting a cancer warning on chocolate cause you can technically eat too much of it.