I was able to make it working using the below algorithm.

private static long findNums(long totalTiles){
    int maxLvel = (int)Math.sqrt(totalTiles-1)/2;
    long count = 0;
    for(int i = 1;i<=maxLvel;i++){
        //Finds number of laminates of the same level. For a a given hole size h and a level l (2*(l+h/2))^2 <= tiles + h^2
        count += (long) ((totalTiles/(4*i))-i);
    }

    return count;
}