Hint: there's a simple recursion relation to determine the number of ways of arranging Y pairs of non-distinguishable curly parentheses, with the number of square parentheses X=0. Let's define "blocks" as top-level parantheses and their contents, for example, for the configuration (()) () (), there are three blocks, the first being (()), and two more ().

Let C(Y) be the number of arrangements of Y pairs of curly parentheses. Clearly, we have C(0)=1, e.g. just the empty arrangement, and C(1)=1, e.g. (). Next, we have C(2)=2, e.g. (()) and () ().

Now let's consider arbitrary Y. The first block can contain Z pairs inside (not counting the top-level pair of the block itself), where Z can be 0,1,2...Y-1. Those Z can have C(Z) arrangements. Once we place Z pairs inside the first block, we will have Y-Z-1 pairs remaining, and those can have C(Y-Z-1) arrangements. Hence, we have

C(Y) = Prod C(Z) C(Y-Z-1);  Z=0,1,2...Y-1

This is exactly the recursion relation for the well-known Catalan numbers.

Also, note that we need the Catalan number for X also, since the square brackets also must be arranged. Since X goes up to 10^5, the recursion above is not fast enough to compute. There is another form for Catalan number involving factorial that can be computed in O(X) time instead of O(X^2).

Some more hints: we also need to compute D[i][j], the number of i-pair parentheses arrangements that consist of exactly j blocks. The recursion relation for this is slightly tricky to derive, but we need only fill out a matrix with O(Y^2) elements.

Finally, note that the number of cases can go up to T~1000, but the computations above need only be done once, and once you have the Catalan numbers, factorials, and matrix elements of D computed, some further simple lookups is as that's needed to get all the answers.