Rabbi Gordon joined his father-in-law as rosh yeshiva at the age of 24, in approximately 1907. While rosh yeshiva, there was a major threat of yeshiva students being drafted into the Polish army, which at the time was known to be anti-Semitic.^[4] He therefore traveled to Warsaw and approached Noach Pryłucki, the dean of Jewish members of the Sejm (the Polish parliament) and an old maskil. Pryłucki told him that the only way his students would be exempt from the draft would be if they added secular studies to the curriculum. Rabbi Gordon didn't not automatically abandon the idea and said that he needed to ask the opinion of greater rabbis. This turned out to be impossible as all communication with Rabbi Chaim Ozer Grodzensky in Vilna was severed during the war. In the end, the mashgiach of the yeshiva, Rabbi Moshe Rosenstain, together with a student, bribed the chairman of a local draft board with American dollars. Rabbi Rosenstain justified this seemingly illegal action, saying that the only reason the yeshiva students were drafted was because of anti-Semitism, as the Christian divinity students in Poland were exempt.^[5]

In 1926, Rabbi Gordon sent fifty students to the Eretz Yisrael where another branch of the yeshiva would be established. Rabbi Lazer Shulevitz, who was living there already, chose to open the yeshiva in Petach Tikvah.^[3] To support his two yeshivos, Rabbi Gordon had to travel throughout the world to collect funds.^[4] Rabbi Reuven Katz, Petach Tikvah's chief rabbi, served as rosh yeshiva alongside Rabbi Gordon.

On September 7, 1939, the Nazi 21st Infantry Division invaded Lomza and the Battle of Łomża broke out. Three quarters of the city were destroyed but the yeshiva building, as well as the Talmud Torah and Rabbi Gordon's house next to the yeshiva, remained standing. Rabbi Gordon had been in America at the time and so Jewish families escaping the bombing in other parts of the city took refuge in his house.^[6]