If you want to improve your security in Office 365 it is recommended to add the EM+S E3 or E5 security suits. This gives you more information about what is happening with your users, but you can configure alerting and actions as well.
So if you have the EM+S licenses, you can go to https://portal.cloudappsecurity.com and start configuring your alerts and policies.
By default there are a lot of default policies, but you can create your own as well! Let me summarize the most important ones that you definitely need to look at:
- Leaked credentials
- Activity from infrequent country
- Unusual file share activity (by user)
- Multiple failed login attempts
- Activity from suspicious IP Addresses
- Suspicious inbox forwarding
- Impossible travel
- Suspicious inbox manipulation rule
Of course there are more to create. But these will help you to mitigate the most important security threats.
After the creation of the policy, you can configure alerting. This can be either by Email, Text message or using flow. You also might want to limit an alerting rule to a specific set of users, let say for example HR and management, so you can Scope the policy.
Last is Governance, in case of a high severity or critical person you might want to suspend the user for follow up.