Find (and delete) duplicate files

I’ve had issues in the past where a Kerio Connect mail server talking to Apple Mail can somehow end up duplicating hundreds or even thousands of emails in mailboxes. I never got to the bottom of what was causing it, but the symptoms were that people would see their mailbox sizes grow to crazy sizes and there would be many, many duplicates of many emails in these folders.

When the issue was occurring more frequently, I was using some command-line tools to clean them up, but having not encountered it in the wild again now for more than a year, I’ve totally forgotten which tools I used and how I invoked them.

In migrating a client’s email to Office 365 recently, one user had a massive mailbox, with one folder alone having over 35 GB of email in it. In searching for the tools I previously used to clean it up, I came across dupeGuru – an OS X GUI application that finds identical files and can trash them.

I haven’t been able to run it head-to-head against whatever I used to use (I think it was fslint but it may have been dupes, or it may have been something else altogether) but whatever, I only need to clean a couple of folders as a one-off task.

Anyway, dupeGuru seems to do the task and runs in a reasonable amount of time, so if you’re looking for an easy to use utility to find and nuke identical files, give it a spin.

Easily create and install Launch Daemons on macOS or OS X

I recently needed to have a script execute on a daily basis – previously this would have been a task for cron however it’s been deprecated (yet still all there) in macOS since something like 10.4.

The accepted way to do it now is via a Launch Daemon. Simple in theory, but easier said than done. It’s far easier to edit a single line in a crontab entry (even if you have to look up the unique syntax of the crontab) than it is to create a Launch Daemon xml .plist file.

Fortunately, Nathan Witmer has created Launched – a web-based tool to create launchd plist files.

From here it was easy – I created my script as per normal and put it in /usr/local/bin

I made up a name for my daemon, set the command to /usr/local/bin/my_amazing_script.sh and configured it to run every day at 8pm (ironically, using the cron syntax to schedule it).

After hitting the Create .plist button, I was given my entire plist to copy and paste, as well as a couple of snippets to install the plist via using curl to get it from Launched, copying it to /Library/LaunchDaemons and using launchctl to load it.

Find Large Items in Office 365 Mailboxes

$UserCredential = Get-Credential

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/\
-Credential $UserCredential -Authentication Basic -AllowRedirection

Import-PSSession $Session

Get-MailboxFolderStatistics -Identity [email protected] -IncludeAnalysis -FolderScope All\ | Select-Object Name,FolderPath,ItemsInFolder,FolderSize,TopSubject,TopSubjectSize,TopSubjectCount,TopSubjectPath | Out-file C:\Users\%username%\Desktop\UserMailboxResults.txt

 

Recursively Propagate Permissions on a Synology NAS with synoacltool

Synology DSM lets you do some pretty complex things with permissions on files and folders, however occasionally things can get a bit mixed up and it’s difficult to see what’s happening and even more difficult to recover from it.

Whilst you can go into the DSM web interface and check the permissions at the top level of each share point in Control Panel > Shared Folder, it’s difficult to see what’s going on below this.

To dig deeper we need to get into the command-line, so you first have to enable ssh access. Go into Control Panel > Terminal & SNMP and Enable SSH service

You can now ssh into the Synology as an admin user (provided your firewall rules on the NAS allow it)

I found that if you modify the permissions on a file or folder with chmod then this seems to wipe out the ACL information. Never mind, there’s another utility called synoacltool that lets you modify ACLs. Unfortunately however synoacltool doesn’t have a switch to operate recursively.

Unix find to the rescue!

What I was able to do however was use find to run it on each and every file and folder in a given folder – this likely isn’t the most efficient way to do it, but you (hopefully) don’t need to do this too often anyway…

First, wipe out the ACLs and set the unix permissions with chmod

chmod -R a+rwX /volume1/Share\ Point

Set the permissions on your share point in DSM and check it in the terminal

ls -ale /volume1/Share\ Point

I got this far, however when I checked the permissions on the files and folders inside Share Point they still had plain ol’ unix permissions and no acls

cd /volume1/Share\ Point

ls -ale

no ACLs. Let’s get them happening.

find . -execdir synoacltool -copy /volume1/Share\ Point {} \;

What this does is find each file and folder and then run the exec command on the given file or folder – so it copies the ACL from our top-level folder onto every single file and folder in the Share Point folder.

There’s quite a lot of overhead in doing it this way, but hopefully you only need to do this once and then the permissions will work…

Higher Quality options to Reduce PDF File Size in Preview.app on macOS / OS X

Apple’s built-in Preview app has long had a way that you can reduce the size of a PDF by recompressing and downsampling the images in the PDF.

Simply open a PDF in Preview and go to File > Export and then for Quartz Filter select Reduce File Size.

Whilst this generally provides very good savings in terms of file size, it is at the expense of the image quality in the PDFs.

Digging a bit deeper into how this is configured, it turns out that the default Quartz Filters are saved in /System/Library/Filters. Knowing that just about everything inside of /System is now off-limits, I created a new folder in /Library called Filters and put some customised versions of the filters in there instead.

After relaunching Preview, my new filters were there to be selected.

Click to download a copy of my quartz filters. It’s pretty self explanatory what they do from their name.

Disable Azure AD users from having to set up a PIN on Windows 10

This information is condensed from Håvard Siegel Haukeberg’s blog over at https://haukeberg.wordpress.com/2016/02/24/disable-pin-code-when-joining-azure-ad/

You need to log in as the actual admin user for the domain, you can’t do this through a partner account with delegated administration.

First up, go to the Azure management portal:

  1. https://manage.windowsazure.com
  2. Go to Active Directory
  3. Select your Domain
  4. Select Applications
  5. Select Microsoft Intune
  6. Select Configure
  7. Under manage devices for these users, select All and click Save.

Next, go to Intune

  1. https://manage.microsoft.com
  2. Go to: Admin > Mobile Device Management > Windows > Passport for Work.
  3. Select: Deactivate Passport for Work on registered devices

 

Give an Azure AD user Local Administrator user privileges on Windows 10

When you join a Windows 10 machine to Azure AD, the user account you use to join to the domain is automatically given local administrator permissions to the machine.

If you noodle around in the Azure management portal, there doesn’t seem to be an easy way to give additional users local administrator permissions on the same machine.

A client of mine is running some practice management software that, as it turns out, requires all users to have local administrator privileges. Yes, I know.

Fortunately there’s an easy to elevate a local user’s account to give them administrator permissions.

Launch a CMD shell with Administrator privileges and type in:

net localgroup Administrators AzureAD\UserName /add

where UserName is their first and last names as specified in Azure; e.g. mine would be KaiHowells

Fix file timestamps with dates in the future on OS X or Unix/Linux

I recently needed to fix a heap of files with invalid timestamps for a client to sync between two sites.

The issue was that some files had modification dates in the year 2032 or something like that. With a 2-way sync, if one file changes the modification date is used to determine which one wins. If the unchanged file has it’s date set in the future, then it could overwrite the changes made that set the other file to the current date and time.

Fortunately fixing this is really easy:

touch /tmp/now
find -xdev /Path/To/Server/Share\ Point -newer /tmp/now -print0 | xargs -0 touch -t 201610131030
rm /tmp/now

I specifically had to set the date and time to an exact date and time (-t 201610131030) rather than letting touch set the files to the current date and time so that they didn’t have time stamps off by a few seconds at each end, causing them to be resynchronised.

The number given to touch is of the format YYYY MM DD HH MM SS. The timestamp is in the local time zone, so I also had to adjust the timestamp given to the files on the remote server as it’s in a different time zone.

Issues authenticating Windows 10 to OS X/macOS with Server.app

Ever since Apple were forced to ditch Samba due to their changing the licensing to GPL3, we’ve been stuck with a second-rate SMB implementation on OS X and Server called smbx.

As it turns out, there are numerous issues with smbx that may or may not end up getting fixed that were pretty much solved problems in Samba. That, and there’s a lot more support out there for Samba across all of the platforms it runs on than there is for smbx.

Oh well, we must make do with what we’ve got…

Windows 10, it seems, has issues connecting to smbx on recent versions of OS X or macOS – something to do with the LanMan compatibility level – i.e. the version of SMB/CIFS that Windows speaks. What it looks like is happening is that Windows is trying to use the older LM or NTLM authentication whereas the Mac wants the newer, and more secure, NTLMv2 authentication method to be used. The result however is that you’re told you have an incorrect username or password.

I’m still looking for a fix on the Mac server side, however I have found a fix that works when applied to the Windows client.

On Windows, open Regedit and go to:

HKLM\System\CurrentControlSet\Control\Lsa

Find the key for LmCompatibilityLevel and change it to 3 (from the default value of 1)

Quit out of Regedit and be amazed that you can now connect to the Mac server.

There is more info over on TechNet if you want to know exactly what this key does.

I haven’t tested to see if denying SMB3 connections at the server end makes any difference…

Troubleshooting: Fetch a web page as Googlebot

I’ve had a couple of clients come to me recently after their WordPress site was pwned. Sometimes you’re able to use a tool like Wordfence to clean it up and secure the site and it’s all OK. Sometimes however it goes a bit deeper than this.

With one hack recently, even after the WordPress core files were cleaned, all plugins updated with fresh versions from the repository and many miscellaneous php files scattered within the wp-content directory were removed, it all looked OK – except when Google indexed the site.

Whilst Google have a utility in their webmaster tools to fetch a page as Google, there are a few limitations. First of all, you need to have the website added to your Google Analytics account (or be logged into a Google Analytics account that owns the property) and it’s not instant.

I was trying to track down something that didn’t show up on any Wordfence scan and wasn’t a malicious plugin or hacked core file. I was quite sure about this as the spam links on the page persisted even after WordPress was reinstalled and all plugins were disabled.

The one thing that did fix it however was switching the theme.

As it turns out, the hackers had inserted some conditional code into the main theme files so that whenever a regular browser was viewing the site, everything looked as it should. When Googlebot viewed the site (actually, any user agent on a long list of user agents used by search engine spiders) there were a huge number of spam links inserted into the page.

I was able to find the snippet of code in one of the theme files and removed it easily, however I found that the Fetch as Google was slow to use in practice. Through using a quick trick with Curl, I could give the website a user agent that triggered the spam links like so:

curl -L -A "Googlebot/2.1 (+http://www.google.com/bot.html)" http://example.com

The above command runs curl with -L (or –location) and -A (or –user-agent) to set the user agent. The Location switch tells curl that if it’s given a redirect code, to send the same request to the new location (i.e. it will send the same user agent to the new location).

This was able to quickly show me the spam content in the page and give me instant feedback that the html output was clean once I had found and removed the suspect chunk of code.