Manually clean up zimbra zmcat/zmcpustat exploit

Symptoms: server load high, cpu usage 300%

To clean up manually follow this steps:

• STOP the process but dont kill it, use
  kill -STOP [PID]
• Hide from search engine (su zimbra):
  zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"
• chmod 755 /opt/zimbra/data/tmp/
  chmod 755 /opt/zimbra/data/tmp/upload
• Delete malicious JSP, find using this command:
  grep -R '(request.getParameter.' /opt/zimbra/mailboxd /opt/zimbra/jetty
  For example: /opt/zimbra/mailboxd/webapps/zimbraAdmin/public/jsp/Alert.jsp and /opt/zimbra/jetty/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
• > /opt/zimbra/mailboxd/webapps/zimbraAdmin/public/jsp/Alert.jsp
  > /opt/zimbra/jetty/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
  chattr +i /opt/zimbra/mailboxd/webapps/zimbraAdmin/public/jsp/Alert.jsp /opt/zimbra/jetty/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
• Fix modified jsp files, find using:
  rpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5' | grep .jsp
  
  For example: /opt/zimbra/jetty-distribution-7.6.12.v20130726/webapps/zimbra/public/login.jsp and /opt/zimbra/jetty-distribution-7.6.12.v20130726/webapps/zimbraAdmin/public/jsp/Debug.jsp
  
  Delete line like this and save: <% if ( "dKpDym-mK4qvvr-YnoG4pFZohbAtQTU9afCr_BPXIOs"
• chattr +i /opt/zimbra/jetty-distribution-7.6.12.v20130726/webapps/zimbra/public/login.jsp /opt/zimbra/jetty-distribution-7.6.12.v20130726/webapps/zimbraAdmin/public/jsp/Debug.jsp
• Find files modified in 15 days ago, fix the file if it is from rpm or delete it:
  find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
  find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
  find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
  
  I deleted this folder: /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/public_  /opt/zimbra/jetty/work/zimbraAdmin/jsp/org/apache/jsp/public_
• Fix crontab, delete line zmcpustater
  crontab -e -u zimbra
• > /opt/zimbra/log/zmcpustat
  > /opt/zimbra/log/zmcpustater
  chattr +i /opt/zimbra/log/zmcpustat /opt/zimbra/log/zmcpustater
• delete executables in /opt/zimbra/log/
  file /opt/zimbra/log/* | grep ELF | awk -F: '{ print $1 }'|xargs rm
• Edit /opt/zimbra/mailboxd/etc/service.web.xml.in (backup first)
  Find tags: ProxyServlet and AutoDiscoverServlet, and remove %%zimbraMailPort%% and %%zimbraMailSSLPort%% in both allowed.ports param
• Restart zimbra !!!
• Now kill the process:
  kill -9 [PID]
• Login to webmail and zimbra admin console, then run this command:
  
  find /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/public_  /opt/zimbra/jetty/work/zimbraAdmin/jsp/org/apache/jsp/public_ -type f |xargs chattr +i  

Ref: https://lorenzo.mile.si/zimbra-cve-2019-9670-being-actively-exploited-how-to-clean-the-zmcat-infection/961/