transform_selection applies a transformation matrix to a selection. The matrix should actually be passed in as a single list -- not a list of lists.
When homogenous == 0: the matrix needed is a 4x4 matrix, where the upper-left 3x3 quadrant is a typical rotation matrix. The 4th row and 4th column specify the pre- and post-translation vectors. So, you can translate+rotate+translate with this one matrix.
cmd.transform_selection("a", [x,x,x,x, x,x,x,x, x,x,x,x, x,x,x,x], homogenous=1)
I wrote a structure alignment program that needs to update the coordinates of the 2nd protein by translating it, rotating it and then translating it on top of the other molecule. It's done as follows:
// // C code // // ...after the magic is done, pack all the values into the TTT matrix. PyObject* pyU = Py_BuildValue( "[f,f,f,f, f,f,f,f, f,f,f,f, f,f,f,f]", bestU, bestU, bestU, bestCOM1, bestU, bestU, bestU, bestCOM1, bestU, bestU, bestU, bestCOM1, -bestCOM2, -bestCOM2, -bestCOM2, 1.);
where bestU is the transpose of the best rotation matrix, bestCOM1 is the center of mass (COM) for molecule 1, and bestCOM2 is for two. pyU becomes rotMat. The call in Python is then,
# # Python code. # cmd.transform_selection( mol2, rotMat, homogenous=0 );
Were I not to use bestCOM1 or bestCOM2, each of those entries would be 0. The last element should always be 1.